Image Fusion Research Articles

Fractional wavelet combined with multi-scale morphology and PCNN hybrid algorithm for grayscale image fusion

Fractional wavelet combined with multi-scale morphology and PCNN hybrid algorithm for grayscale image fusion

Multispectral Pedestrian Detection Based on Prior-Saliency Attention and Image Fusion

Detecting pedestrians in varying illumination conditions poses a significant challenge, necessitating the development of innovative solutions. In response to this, we introduce Prior-AttentionNet, a pedestrian detection model featuring a Prior-Attention mechanism. This model leverages the stark contrast between thermal objects and their backgrounds in far-infrared (FIR) images by employing saliency attention derived from FIR images via UNet. However, extracting salient regions of diverse scales from FIR images poses a challenge for saliency attention. To address this, we integrate Simple Linear Iterative Clustering (SLIC) superpixel segmentation, embedding the segmentation feature map as prior knowledge into UNet’s decoding stage for comprehensive end-to-end training and detection. This integration enhances the extraction of focused attention regions, with the synergy of segmentation prior and saliency attention forming the core of Prior-AttentionNet. Moreover, to enrich pedestrian details and contour visibility in low-light conditions, we implement multispectral image fusion. Experimental evaluations were conducted on the KAIST and OTCBVS datasets. Applying Prior-Attention mode to FIR-RGB images significantly improves the delineation and focus on multi-scale pedestrians. Prior-AttentionNet’s general detector demonstrates the capability of detecting pedestrians with minimal computational resources. The ablation studies indicate that the FIR-RGB+ Prior-Attention mode markedly enhances detection robustness over other modes. When compared to conventional multispectral pedestrian detection models, Prior-AttentionNet consistently surpasses them by achieving higher mean average precision and lower miss rates in diverse scenarios, during both day and night.

Navigating the evolving diagnostic and therapeutic landscape of low- and intermediate-risk prostate cancer

Abstract In this nonsystematic review of the literature, we explored the changing landscape of detection and treatment of low- and intermediate-risk prostate cancer (PCa). Through emphasizing improved cancer assessment with histology classification and genomics, we investigated key developments in PCa detection and risk stratification. The pivotal role of prostate magnetic resonance imaging (MRI) in the novel diagnostic pathway is examined, alongside the benefits and drawbacks of MRI-targeted biopsies for detection and tumor characterization. We also delved into treatment options, particularly active surveillance for intermediate-risk PCa. Outcomes are compared between intermediate- and low-risk patients, offering insights into tailored management. Surgical techniques, including Retzius-sparing surgery, precision prostatectomy, and partial prostatectomy for anterior cancer, are appraised. Each technique has the potential to enhance outcomes and minimize complications. Advancements in technology and radiobiology, including computed tomography (CT)/MRI imaging and positron emission tomography (PET) fusion, allow for precise dose adjustment and daily target monitoring with imaging-guided radiotherapy, opening new ways of tailoring patients’ treatments. Finally, experimental therapeutic approaches such as focal therapy open new treatment frontiers, although they create new needs in tumor identification and tracking during and after the procedure.

Improving spatiotemporal image fusion incorporating unmixing step by considering the point spread function effect

ABSTRACT In this paper, an improving spatiotemporal image fusion (STIF) incorporating unmixing step by considering the point spread function (PSF) effect model (CPSF) is proposed to address the problem that unmixing accuracy in STIF models incorporating unmixing step is vulnerable to the PSF effect. First, an unsupervised clustering algorithm is used to cluster the fine image at previous date into a land cover map, and the coarse class fraction matrix is generated from this land cover map. Second, the fine class fraction matrix uncontaminated by the PSF effect is calculated by upsampling the coarse class fraction matrix using the Area-to-Point Kriging (ATPK) method under the PSF constraint. Third, this fine class fraction matrix is convolved with the ideal square wave filter to downsample, and then normalized to obtain the coarse class fraction matrix without the influence of the PSF effect. Finally, the CPSF-processed class fraction matrix is used to replace the original one in the following unmixing and prediction process to generate the fine resolution image at the prediction date. Experiments on two remote sensing data sets show that the proposed method can effectively improve the accuracy of spatiotemporal image fusion methods incorporating unmixing step.

Infrared and visible image fusion method based on visual saliency objects and fuzzy region attributes

Infrared and visible image fusion method based on visual saliency objects and fuzzy region attributes

Multispectral and hyperspectral images fusion based on subspace representation and nonlocal low-rank regularization

ABSTRACT Multispectral image (MSI) and hyperspectral image (HSI) fusion is a popular method for HSI super-resolution reconstruction (HSI-SR). MSI-HSI fusion problem is ill-posed and demands several image priors or regularization terms to solve accurately, which is a challenging issue. In this paper, we propose an MSI-HSI fusion model via subspace representation and nonlocal low-rank regularization (SRNLRR). SRNLRR model incorporates the global spectral correlations and spatial nonlocal similarities of HSI to improve the fusion results, where the priors complement each other. First, we use the mode-n tensor-matrix product to project latent high spatial resolution HSI (HR-HSI) into spectral subspace, which can capture spectral low-rankness and reduce computational complexity. Then, based on low-rank representation (LRR) and nonlocal processing strategy, we design a spatial nonlocal LRR regularization (spa-NLRR) and a spectral global LRR regularization (spe-GLRR). These two regularizations analyze the spatial nonlocal similarities and spectral global correlations from intermediate-level vision. Finally, we use the residual regularization program to obtain more image information and input it into the fusion model. We use alternate minimization (AM) methods to optimize the SRNLRR model and employ the alternating direction method (ADM) on spatial/spectral LRR learning. Comparison experiments between the SRNLRR method and six advanced methods on three HSI datasets indicate the superiority of our method.

Abstract PO4-07-08: Impact of an MRI/US fusion imaging for preoperative planning using non-wire localization technique in breast conserving surgery in patients with non-mass enhancement on MRI. A prospective multicenter study

Abstract [Background] Preoperative breast MRI is used to evaluate for additional and extent disease for newly diagnosed breast cancer patients with dense breasts. The potential benefit of preoperative MRI is still being controversial. However, when breast conserving surgery (BCS) is considered for non-mass enhancement (NME) on MRI, the precise image-guided localization is required. Although breast ultrasound (US) plays an adjunctive role in evaluation of MRI findings in preoperative planning for BCS, NME is less likely than a mass or focus to have a US correlate. Furthermore, the presence of NME increases the re-operation rate in BCS because of the difficulty in evaluating the tumor margin. An MRI/US fusion imaging has recently been developed that uses either real-time virtual sonography TM (RVS) or volume navigation TM (Vnav), and can overlay an US with the MRI image of the same site in real time by using magnetic position tracking system. The purpose of this prospective multicenter study was to evaluate the effect of MRI/US fusion imaging for preoperative planning using non-wire localization technique in BCS in patients with NME. [Methods] In 3 Japanese hospital, 13 patients who had lesions with NME that exceeded the US hypoechoic area were enrolled in this study from 2019 to 2022. Written informed consent was acquired, and then an additional supine MRI using a body surface coil was performed. During preoperative planning before BCS, RVS/Vnav was used to determine the NME enhancing area and to mark the area on the skin. Tissue markers were inserted under US at a location assessed by RVS/Vnav as the edge of extent of NME. We analyzed both the surgical margin positivity rate and the reoperation rate. [Results] The preoperative diagnosis by core needle biopsy were 6 DCIS and 7 invasive ductal carcinoma. NME distribution types were 7 segmental, 3 focal, 2 linear 4 and 1 regional. The median diameters of the NME and hypo-echoic lesions were 31 mm (range: 18–39 mm) and 16 mm (range: 9–32 mm), respectively (p = 0.0005). After confirming the localization of the tissue marker with intraoperative US, 12 lumpectomy and 1 quadrantectomy were conducted. The median specimen weight was 63g (range: 54-105g). In the final pathologic diagnosis of excised specimens were 5 DCIS and 8 IDC. In 8 cases (62%), the pathologic diameter was larger than the US diameter. All surgical margins were negative. None of the patients required additional resection. [Conclusions] The findings of study suggest that preoperative planning using non-wire localization technique under MRI/US fusion imaging for BCS in patients with NME could improve both a surgical margin positive rate and a reoperation rate. Citation Format: Shogo Nakano, Manabu Futamura, Keitaro Kamei, Takayoshi Uematsu, Miwa Yoshida, Sadako Akashi-Tanaka, Junta Sakakibara, Ichiro Isomoto, Hiroko Satake, Maiko Takahashi, Minami Hatono, Junko Araki. Impact of an MRI/US fusion imaging for preoperative planning using non-wire localization technique in breast conserving surgery in patients with non-mass enhancement on MRI. A prospective multicenter study [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PO4-07-08.

SCAKD: a knowledge distillation framework based on spatial-corner attention for infrared and visible image fusion

SCAKD: a knowledge distillation framework based on spatial-corner attention for infrared and visible image fusion

Fusion Imaging Guidance for Coronary Reaccess After Valve-in-Valve Transcatheter Aortic Valve Replacement

Coronary reaccess can be difficult after valve-in-valve transcatheter aortic valve replacement. We present a case whereby prior valve-in-valve transcatheter aortic valve replacement created difficulty with coronary reaccess. Computed tomography-fluoroscopy fusion imaging was used for guidance to successfully engage the coronary arteries and perform percutaneous coronary intervention.

Rapid determination of residual pefloxacin in mutton based on hyperspectral imaging and data fusion

Non-destructive detection of veterinary drug residues in meat products is critical for ensuring food safety and human health. The feasibility of using visible near-infrared (Vis-NIR) and near-infrared (NIR) hyperspectral imaging (HSI) system combined with data fusion for the prediction of pefloxacin residues in mutton was investigated in this study. A partial least squares regression optimization (PLSR) model was developed by multivariate data processing methods. The results showed that the low-level fusion (LLF) produced better results with R2P = 0.907 and root mean square error of prediction (RMSEP) = 0.462 as compared to individual data blocks. Furthermore, intermediate-level fusion (ILF) based on iterative retention of informative variables (IRIV) algorithm showed the best results with R2P = 0.940 and RMSEP = 0.375. Finally, the visualized distributions of Vis-NIR and NIR based on the optimal model combination were mapped. The results have demonstrated that a combination of HSI with data fusion may be applied to rapidly and non-destructively detect pefloxacin residues in mutton.

Edge preserving infrared and visible image fusion with three layer decomposition based on multi-level co-occurrence filtering

Edge preserving infrared and visible image fusion with three layer decomposition based on multi-level co-occurrence filtering

Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

An unsupervised multi-focus image fusion method via. dual-channel convolutional network and discriminator

The challenge in multi-focus image fusion tasks lies in accurately preserving the complementary information from the source images in the fused image. However, existing datasets often lack ground truth images, making it difficult for some full-reference loss functions (such as SSIM) to effectively participate in model training, thereby further affecting the performance of retaining source image details. To address this issue, this paper proposes an unsupervised dual-channel dense convolutional method, DCD, for multi-focus image fusion. DCD designs Patch processing blocks specifically for the fusion task, which segment the source image pairs into equally sized patches and evaluate their information to obtain a reconstructed image and a set of adaptive weight coefficients. The reconstructed image is used as the reference image, enabling unsupervised methods to utilize full-reference loss functions in training and overcoming the challenge of lacking labeled data in the training set.Furthermore, considering that the human visual system (HVS) is more sensitive to brightness than color, DCD trains the dual-channel network using both RGB images and their luminance components. This allows the network to focus more on the brightness information while preserving the color and gradient details of the source images, resulting in fused images that are more compatible with the HVS. The adaptive weight coefficients obtained through the Patch processing blocks are also used to determine the degree of preservation of the brightness information in the source images.Finally, comparative experiments on different datasets also demonstrate the superior performance of DCD in terms of fused image quality compared to other methods.

CABnet: A channel attention dual adversarial balancing network for multimodal image fusion

CABnet: A channel attention dual adversarial balancing network for multimodal image fusion

Deep M2CDL: Deep Multi-Scale Multi-Modal Convolutional Dictionary Learning Network.

For multi-modal image processing, network interpretability is essential due to the complicated dependency across modalities. Recently, a promising research direction for interpretable network is to incorporate dictionary learning into deep learning through unfolding strategy. However, the existing multi-modal dictionary learning models are both single-layer and single-scale, which restricts the representation ability. In this paper, we first introduce a multi-scale multi-modal convolutional dictionary learning ( M2CDL) model, which is performed in a multi-layer strategy, to associate different image modalities in a coarse-to-fine manner. Then, we propose a unified framework namely Deep M2CDL derived from the M2CDL model for both multi-modal image restoration (MIR) and multi-modal image fusion (MIF) tasks. The network architecture of Deep M2CDL fully matches the optimization steps of the M2CDL model, which makes each network module with good interpretability. Different from handcrafted priors, both the dictionary and sparse feature priors are learned through the network. The performance of the proposed Deep M2CDL is evaluated on a wide variety of MIR and MIF tasks, which shows the superiority of it over many state-of-the-art methods both quantitatively and qualitatively. In addition, we also visualize the multi-modal sparse features and dictionary filters learned from the network, which demonstrates the good interpretability of the Deep M2CDL network.

Diff-IF: Multi-modality image fusion via diffusion model with fusion knowledge prior

Diff-IF: Multi-modality image fusion via diffusion model with fusion knowledge prior

Privacy-preserving federated transfer learning for defect identification from highly imbalanced image data in additive manufacturing

Defect identification is a crucial task for process monitoring and quality evaluation in additive manufacturing (AM). Deep learning (DL) has shown great potential for diverse fields, but some challenges have hindered the application in AM process monitoring. Firstly, DL-based methods are driven by big data and require a large number of training data. However, in reality, defective data is often rare, so different AM manufacturers can collaborate to train a global model for detecting defects. Nevertheless, due to privacy concerns and conflicts of interest, data-sharing between different manufacturers might be dangerous. Additionally, the heterogeneities among manufacturers’ data leads to the domain shift, making it difficult to obtain a well-generalized model. Moreover, the imbalance issue of powder-spreading defects seriously damages the performance of the defect recognition. In this paper, we proposed FTLAM, a federated transfer learning method, to address the above issues. Concretely, to solve the data privacy concerns, federated learning (FL) is utilized to collaboratively train a global model between different AM manufacturers without exchanging original data. Furthermore, a client-oriented transfer method is proposed to mitigate the heterogeneities across multiple clients and improve the model's generalization. Meanwhile, a multi-loss joint optimization approach is designed to alleviate the data imbalance. Extensive experiments on laser powder bed fusion (LPBF) image datasets of powder-spreading have demonstrated that our FTLAM can obtain satisfactory performance of powder spreading defect identification.

Multimodal image fusion on ECG signals for congestive heart failure classification

Multimodal image fusion on ECG signals for congestive heart failure classification

Measurement method for live chicken shank length based on improved ResNet and fused multi-source information

Phenotypic parameters are crucial reference indicators in poultry breeding. However, the chickens shank length is still manually measured, which is time-consuming and labor-intensive. Additionally, the measurement results are difficult to unify due to the subjective factors of different individuals. To address this issue, this paper proposed a method for live chicken shank length measurement (SLM). It enriches chicken shank feature by fusing visible images and infrared images. The fusion images are then input into a deep regression model based on the improved ResNet. The measurement model used ResNet as its backbone and introduces Squeeze-and-Excitation (SE) blocks and a Spatial Pyramid Pooling (SPP) block, resulting in more precise and stable shank length measurements. The average coefficient of variation, average floating error, average standard deviation and Pearson correlation coefficient for shank length measurements using the fusion images are 0.21 %, 0.49 %, 0.181 mm and 0.996, respectively, compared with using single visible or infrared image, the accuracy and stability are obviously improved. That indicated combining deep learning model and fusion information, the SLM proposed in this paper can achieve a more precise, reliable and standardized measurement of live chicken shank length.

LRFE-CL: A self-supervised fusion network for infrared and visible image via low redundancy feature extraction and contrastive learning

Aiming to amalgamate the distinct yet complementary attributes inherent to infrared and visible spectra, Infrared and Visible Image Fusion (IVIF) endeavors to preserve an optimal intensity distribution while curtailing the superfluous data within the resultant image. Nonetheless, it is an incontrovertible reality that the majority of extant methods are prone to engendering an undesirable redundancy of features, in addition to a marked degradation throughout the fusion process. To address the issue, we proposed a self-supervised fusion network for infrared and visible image via low redundancy feature extraction and contrastive learning, termed as LRFE-CL. Specifically, the Low Redundancy Feature Extraction Network (SSFN-LRAE) has been developed to effectively extract low-redundancy features from infrared and visible images using the self-supervised strategy and the Channel Shift Attention Module (CSAM). Subsequently, the Contrastive Fusion Decoder (CFDN) is designed to fuse multi-layer features extracted from the SSFN-LRAE. To prevent feature degradation, a novel regularized contrastive loss mechanism targeting multi-layer features has been formulated to optimize our CFDN, ensuring the fusion of comprehensive texture details and brightness information. Our comprehensive experimental results unequivocally demonstrate the exceptional performance of our method, both in terms of visual effects and fusion performance.